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DETAILED ACTION 
Response to Amendment 

1 . The amendment filed on 1 1/16/2006 has been entered. The amendments to the 
claims appears to be maintaining a line of demarcation between the 09/818,388, now 

% US patent no. 7,176,936, and this application, thus, the double patenting warning is 
overcome. The amendments to the claims also overcome the 35 USC 112 first 
paragraph rejection but do not clearly claim the invention as discussed below under 35 
USC 112 second paragraph. 

Response to Arguments 

2. Applicant's arguments filed 11/1 6/2006 have been fully considered but they are 
not persuasive. 

The arguments concerning applicants admitted prior art are not persuasive 
because applicants characterization of the prior art discussed in the specification at 
pages 1-5 as being limited to fluror-guided medical procedures (i,e, X-ray procedures) is 
unduly limiting since the pages 1-5 discuss prior art systems which are not limited to 
fluror-guided medical procedures and which are similar to applicants system which use 
the same see-through head mounted display or video-see-through head mounted 
display that applicant uses, see specification at page 44 lines19-22, to merge computer 
generated graphical path guides and target with actual instrument and actual object. 
Additionally the specification at pages 34-36 compares the prior art bulls eye with 
applicants bulls eye, however, both methods merge a computer generated bulls eye 
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and target onto the view of the actual instrument and actual object. Therefore, the 
admitted prior art teaches the rejected claims. Thus, the claims need to be amended to 
claim a system that does not read on the disclosed prior art method for facilitating the 
user's instrument positioning by using augmented reality. 

The arguments concerning Kienzle at pages 11-13 are not persuasive because 
the claims do not claim how the real instrument and real object is displayed to the user 
due to the word "contains" and in view of steps 735, 740, 750 of figure 7 and in view of 
step 2308 of figure 23 where applicant displays the real instrument graphically. Thus, 
the graphical display of the X ray image with the graphical representation of the real 
instrument along with the graphical representation of the target and the path teaches 
the claim. 

Claim Rejections - 35 USC §112 

3. The following is a quotation of the second paragraph of 35 U.S.C. 112: 

The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 

4. Claims 13, 16-23, 25, 26, 29-35, and 37-43 are rejected under 35 U.S.C. 1 12, 
second paragraph, as being indefinite for failing to particularly point out and distinctly 
claim the subject matter which applicant regards as the invention. 

Claim 13: 

At lines 7-8 this claim claims "displaying the rendered at least one graphics path 
guide on a display overlaid onto a view that contains said real instrument and a real 
object". The same limitation is present at lines 11-12. This claim limitation does not 
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claim the type of display, actual or within the computer such as rendering, and does not 
claim how the real instrument and real object is displayed to the user with the virtual 
instrument and virtual target point due to the word "contains" and in view of steps 735, 
740, 750 of figure 7, in view of step 2308 of figure 23 (step 2308 is described at page 50 
lines 14-17), in view of page 45 lines 7-9, and in view of page 53 lines 10-19 and the 
claimed display is not limited to the display at page 44 line 16 to page 45 line 26. It is 
also not clear whom or what performs the "aligning the real instrument with the path" at 
line 14 and "moving said real instrument along the path" at line 17. At line 1 1 "a display" 
is claimed and at lines 7-8 "a display" is also claimed, thus, making line 11 indefinite 
since line 1 1 does not clearly refer to the display of lines 7-8 or to a new display. 
Claim 16: 

This claim has similar problems that claim 13 has due to substantially similar 
claim amendments and the discussion of claim 13 is modified to more closely correlate 
to the claim language found in this claim. At lines 5-8 this claim claims "rendering a 
graphical representation ...on a display overlaid onto a view that contains said real 
instrument and a real object". The same limitation is present at lines 17-18. This claim 
limitation does not claim the type of display, actual or within the computer such as 
rendering, and does not claim how the real instrument and real object is displayed to the 
user with the virtual instrument and virtual target point due to the word "contains" and in 
view of steps 735, 740, 750 of figure 7, in view of step 2308 of figure 23 (step 2308 is 
described at page 50 lines 14-17), in view of page 45 lines 7-9, and in view of page 53 
lines 10-19 and the claimed display is not limited to the display at page 44 line 16 to 
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page 45 line 26. It is also not clear whom or what performs the "aligning the actual 
instrument to the actual path" at line 14. At line 17 "a display" is claimed and at line 7 "a 
display" is also claimed, thus, making line 17 indefinite since line 17 does not clearly 
refer to the display of line 7 or to a new display. 
Claim 23: 

This claim has similar problems that claim 13 has due to substantially similar 
claim amendments and the discussion of claim 13 is modified to more closely correlate 
to the claim language found in this claim. At lines 7-9 this claim claims "rendering a 
graphical representation ...to obtain a virtual instrument and a virtual target point on a 
display overlaid onto a view that contains said real instrument and a real object". At line 
17 the virtual instrument and virtual target point are displayed and at lines 21-24 the 
graphical representation of the actual target point and the actual path are viewed in the 
form of a graphical target point and at least one graphical axis marker on a display 
along with the view of an actual scene along the actual path but it is not clear if this is 
the same displaying found at lines 7-9 that contains the real instrument and real object. 
This claim limitation does not claim the type of display, actual or within the computer 
such as rendering, and does not claim how the real instrument and real object is 
displayed to the user with the virtual instrument and virtual target point due to the word 
"contains" and in view of steps 735, 740, 750 of figure 7, in view of step 2308 of figure 
23 (step 2308 is described at page 50 lines 14-17), in view of page 45 lines 7-9, and in 
view of page 53 lines 10-19 and the claimed display is not limited to the display at page 
44 line 16 to page 45 line 26. At lines 23-24 "a display" is claimed and at lines 8-9 "a 
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display" is also claimed, thus, making lines 23-24 indefinite since lines 23-24 does not 
clearly refer to the display of lines 8-9 or to a new display. 

Claim Rejections - 35 USC § 102 

5. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 1 02 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
States. 

6. Claims 13, 16-20, 23, 25, 26, 29, 30, 35, 37, 39, 41, 42, and 43 are rejected 
under 35 U.S.C. 102(b) as being anticipated by applicants admission of the prior art. 
See applicants specification at pages 1-5 and page 34 line 16 to page 36 line 8. 
Applicants admission discusses using several types of graphical markers in the 
augmented reality display to allow the user to align the actual instrument with the path 
needed to be taken to place the instrument onto the actual target. 

Claim 13: 

Applicants admission of the prior art bulls eye graphical marker teaches a 
method for augmented reality guided instrument positioning, comprising the steps of: 

rendering at least one graphics path guide for indicating a path for a real 
instrument to follow to a target, the graphics path guide being constructed in a way that 
it frames the path so as not to obstruct a view of a central part of the real instruments if 
the real instruments is in correct alignment to said path (Bulls eye.); 
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displaying the rendered at least one graphics path guide on a display overlaid 
onto a view that contains said real instrument and a real object, which includes said 
target (display 124, page 34 line 16 to page 36 line 8); 

moving said real instrument by a user in response to viewing the displayed at 
least one graphic path guide on a display overlaid onto a view that contains said real 
instrument and a real object, which includes said target to align it with the at least one 
graphics path guide, {page 34 line 16 to page 36 line 8), 

aligning the real instrument with the path by determining when the at least one 
graphics path guide frames the path so that a view of a central part of the real 
instrument is not obstructed by the at least one graphics path guide (Bulls eye.); and 

moving said real instrument along the path so that a front portion of said 
real instrument is inserted into the object until its tip reaches said target (page 34 line 16 
to page 36 line 8). 

Claim 16: 

Applicants admission of the prior art bulls eye graphical marker teaches a 
method for augmented reality guided instrument positioning, comprising the steps of: 

defining a point on an actual target (page 34 line 16 to page 36 line 8); 

defining an actual path to reach the point on the actual target (page 34 line 16 to 
page 36 line 8); 

rendering a graphical representation of the actual target point and the actual path 
in the form of a graphical target point and at least one graphical axis marker on a 
display overlaid onto a view that contains said real instrument and a real object, which 
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includes said target, respectively, the graphical representation being rendered with 
respect to a user's augmented reality viewpoint, wherein the augmented reality view 
combines a view of an actual scene with said graphical representation and the 
augmented reality line of sight to the graphical target point coincides with the actual 
path that the actual instrument needs to be aligned to during a positioning of the actual 
instrument to the actual target point (page 34 line 16 to page 36 line 8); and 

aligning the actual instrument to the actual path by aligning it to the augmented 
reality line of sight towards the graphical target point, and moving the actual instrument 
by a user in response to viewing the rendered graphical representation on a display 
overlaid onto a view that contains said real instrument and a real object, which includes 
said target along the actual path towards the actual target point while keeping it aligned 
with the augmented reality line of sight (page 34 line 16 to page 36 line 8). 

Claim 17: 

Inherently the actual instrument includes at least one physical axis marker for 
alignment with said at least one graphical axis marker. 
Claim 18: 

Inherently the actual instrument includes at least one physical axis marker for 
alignment with said at least one graphical axis marker. 
Claim 19: 

Page 34 line 16 to page 36 line 8 teaches a user makes the augmented reality 
line of sight coincide with the actual path by moving into a position where said graphical 
target point and said at least one graphical axis marker line up. 



Application/Control Number: 09/818,427 Page 9 

Art Unit: 2628 

Claim 20: 

Page 34 line 16 to page 36 line 8 teaches the at least one identified graphical 
axis marker has a circular shape, and is centered on the axis of the actual path. 
Claim 23: 

Applicants admission of the prior art bulls eye graphical marker teaches a 
method for virtual reality guided instrument positioning, comprising the steps of: 

defining a point on an actual target {page 34 line 16 to page 36 line 8); 

defining an actual path to reach the point on the actual target {page 34 line 16 to 
page 36 line 8); 

tracking a pose of an actual instrument with respect to a pose of the actual target 
{page 34 line 16 to page 36 line 8); 

rendering a graphical representation of the actual instrument and the actual 
target point to obtain a virtual instrument and a virtual target point on a display overlaid 
onto a view that contains said real instrument and a real object, which includes said 
target, respectively, the graphical representation being rendered with respect to a virtual 
viewpoint from which a virtual line of sight to the virtual target point coincides with a 
virtual path for the virtual instrument to follow during a positioning of the actual 
instrument to the point on the actual target, the virtual path corresponding to the actual 
path, the virtual instrument comprising a 3D structure for line of sight alignment, the 3D 
structure comprising a plurality of markers centered on and distributed along an axis of 
the virtual instrument {page 34 line 16 to page 36 line 8); 
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displaying the rendered virtual instrument and virtual target point (display 124, 
page 34 line 16 to page 36 line 8); 

aligning the virtual instrument along the virtual line of sight to the virtual target 
point in order to accordingly align the actual instrument along the actual path {page 34 
line 16 to page 36 line 8); and 

moving the actual instrument by a user in response to viewing the rendered 
graphical representation of the actual target point and the actual path in the form of a 
graphical target point and at least one graphical axis marker on a display along with the 
view of an actual scene along the actual path towards the actual target keeping the 
correct alignment by observing and keeping the alignment of virtual instrument and 
virtual path {page 34 line 16 to page 36 line 8). 

Claim 25: 

Page 34 line 16 to page 36 line 8 teaches the virtual target point has a circular 
shape (a center of a target implies a circular target, page 35 line 7.). 
Claim 26: 

Page 34 line 16 to page 36 line 8 teaches wherein the circular shape is a ring. 
Claim 29: 

Page 34 line 16 to page 36 line 8 teaches said plurality of markers comprise at 
least two rings, centered on an axis of the virtual instrument. 
Claim 30: 

The bulls eye of page 34 line 16 to page 36 line 8 teaches the two rings have 
different diameters. 
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Claim 35: 

Page 34 line 16 to page 36 line 8 teaches wherein the rendering step further 
comprises the step of rendering graphical information about a distance between the 
actual instrument and the point on the actual target, the graphical information about the 
distance being overlaid onto the graphical representation. 

Claim 37: 

Page 34 line 16 to page 36 line 8 teaches wherein the virtual target point and the 
virtual instrument are designed such that information corresponding to the distance 
between the actual instrument and the point on the actual target can be directly 
observed from an alignment of the virtual target point and the virtual instrument. 

Claim 39: 

Page 34 line 16 to page 36 line 8 teaches wherein the virtual target point and the 
virtual instrument are each comprised of at least one ring centered on the target point 
respectively on the axis of the instrument, and a diameter of the at least one ring is 
dimensioned to achieve a pre-defined configuration together with the actual instrument 
when the actual instrument reaches the actual target. 

Claim 41: 

Page 34 line 16 to page 36 line 8 teaches wherein said graphical representation 
from the virtual viewpoint is combined with an augmented reality view. 
Claim 42: 

Page 34 line 16 to page 36 line 8 teaches wherein the graphical target point and 
the graphical axis marker are designed such that information corresponding to the 
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distance between the actual instrument and the point on the actual target can be directly 
observed from an alignment of the graphical target point and the graphical axis marker. 
Claim 43: 

Page 34 line 16 to page 36 line 8 teaches wherein the graphical target point and 
the graphical axis marker are each comprised of at least one ring centered on the target 
point respectively on the axis of the instrument, and a diameter of the at least one ring is 
dimensioned to achieve a pre-defined configuration together withy the actual instrument 
when the actual instrument reaches the actual target. 

7. Claims 16, 22, 23, 31-35, 37, and 40-42 are rejected under 35 U.S.C. 102(b) as 
being anticipated by Kienzle, III. US PGPub application no. 2002/ 0077540. 
Claim 16: 

Kienzel teaches a method for augmented reality guided instrument positioning, 
comprising the steps of: 

defining a point on an actual target (figure 7, 1 59); 

defining an actual path to reach the point on the actual target (156); 

rendering a graphical representation of the actual target point (1 59) and the 
actual path (156) in the form of a graphical target point and at least one graphical axis 
marker (156 or 131 or 21 1 ) on a display overlaid onto a view that contains said real 
instrument and a real object, which includes said target, respectively, the graphical 
representation being rendered with respect to a user's augmented reality viewpoint, 
wherein the augmented reality view combines a view of an actual scene with said 
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graphical representation and the augmented reality line of sight to the graphical target 
point coincides with the actual path that the actual instrument needs to be aligned to 
during a positioning of the actual instrument to the actual target point (figures 4-7, 1 1 , 
and 12); and 

aligning the actual instrument to the actual path by aligning it to the augmented 
reality line of sight towards the graphical target point, and moving the actual instrument 
by a user in response to viewing the rendered graphical representation on a display 
overlaid onto a view that contains said real instrument and a real object, which includes 
said target along the actual path towards the actual target point while keeping it aligned 
with the augmented reality line of sight (The user moves the instrument in response to 
viewing the display which shows the alignment of the instrument with the required 
path.). 

Claim 22: 

Kienzel teaches the method according to claim 16, wherein the at least one 
identified graphical axis marker comprises at least two axis markers for controlling 
alignment of the actual instrument along a line of sight (see figures 4-7, 1 1 , and 12 ). 

Claim 23: 

This claim is similar to claim 16 and it is rejected for the same reasons. 
Claim 31: 

Kienzel teaches the method of claim 23, wherein the step of aligning the virtual 
instrument further comprises the step of choosing an orientation of the graphical 
representation around the virtual line of sight according to a pose of a user with respect 
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to the actual target (by sensing the orientation of the real instrument the system has 
sensed the pose of the user). 
Claim 32: 

Kienzel teaches the method of claim 31 , further comprising the step of 
determining the orientation such that east, west, north, and south correspond to right, 
left, forward, and backward, respectively, for the pose of the user in which the user 
faces the actual target, said determining step based on a selection (by sensing the 
orientation of the real instrument the system has sensed the pose of the user). 

Claim 33: 

Kienzel teaches the method of claim 31 , wherein the orientation is dynamically 
adjusted according to a change of the pose of the user (by sensing the orientation of the 
real instrument the system has sensed the pose of the user). 

Claim 34: 

Kienzel teaches the method of claim 32, wherein the selection is dynamically 
adjusted with respect to the pose of the user (by sensing the orientation of the real 
instrument the system has sensed the pose of the user). 

Claim 35: 

Kienzel teaches the method of claim 23 wherein the rendering step further 
comprises the step of rendering graphical information about a distance between the 
actual instrument and the point on the actual target, the graphical information about the 
distance being overlaid onto the graphical representation by graphically displaying the 
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instrument as it is moved by the user relative to the target which shows the distance 
graphically. 

Claim 37: 

Kienzel teaches the method of claim 23 wherein the virtual target point and the 
virtual instrument are designed such that information corresponding to the distance 
between the actual instrument and the point on the actual target can be directly 
observed from an alignment of the virtual target point and the virtual instrument by 
graphically displaying the instrument as it is moved by the user relative to the target 
which shows the distance graphically. 

Claim 40: 

Kienzel teaches the method of claim 23, wherein the graphical representation 
from the virtual viewpoint is combined with another graphical representation from 
another virtual viewpoint looking at the virtual path from a side thereof with reference to 
figures 4-7, 11, and 12. 

Claim 41: 

Kienzel teaches the method of claim 23, wherein said graphical representation 
from the virtual viewpoint is combined with an augmented reality view with reference to 
figures 4-7, 11, and 12. 

Claim 42: 

Kienzel teaches the method of claim 16 wherein the graphical target point and 
the graphical axis marker are designed such that information corresponding to the 
distance between the actual instrument and the point on the actual target can be directly 
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observed from an alignment of the graphical target point and the graphical axis marker 
by graphically displaying the instrument as it is moved by the user relative to the target 
which shows the distance graphically. 



Claim Rejections - 35 USC § 103 

8. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

9. Claim 21 is rejected under 35 U.S.C. 103(a) as being unpatentable over Kienzle, 
III. US PGPub application no. 2002/ 0077540 and applicants admission of the prior art 
and motivation of using aiming aids of guns with instrument guidance found at page 38 
lines 7-16. Kienzel does not teach the graphical axis marker comprises an intersection 
of at least two lines, the intersection to be centered on the axis of the actual instrument 
for correct alignment. 

It would have been obvious to one of ordinary skill in the art to use the feature of 
gun aiming, the cross, and apply it to augmented reality aiming of real instruments 
because this will give the user a good idea of the alignment of the real instruments with 
the real target. 
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10. Claim 38 is rejected under 35 U.S.C. 103(a) as being unpatentable over Kienzle, 
III. US PGPub application no. 2002/ 0077540. Kienzel does not teach rendering by 
using a virtual camera with a wide angle lens. Page 50 line 24 to page 51 line 4 of 
applicants specification does not give a reason for rendering by using a virtual camera 
with a wide angle lens. 

It would have been obvious to one of ordinary skill in the art to render by using a 
virtual camera with a wide angle lens because this is one of many ways to render the 
graphical images and because applicant did not give any reason for choosing this type 
of rendering. 

1 1 . THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 . 1 36(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 

12. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jeffery A Brier whose telephone number is (571 ) 272- 
7656. The examiner can normally be reached on M-F from 7:00 to 3:30. If attempts to 
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reach the examiner by telephone are unsuccessful, the examiner's supervisor, Michael 
Razavi, can be reached at (571 ) 272-7664. The fax phone Number for the organization 
where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 




Jeffery A Brier 
Primary Examiner 
Division 2628 



